I. Field of the Invention
The present invention relates generally to video processing, and more specifically to edge adaptive interpolation of Bayer patterns.
II. Description of the Related Art
The increase popularity of digital cameras for still images and motion pictures has resulted in great advances in digital imaging and video processing. An active area of investigation is in CFA (color filter array) recovery methods. In some digital cameras such as those using charge coupled device (CCD) sensors or Metal Oxide Semiconductor (CMOS) sensors, a single sensor is used to sub-sample an image into three color planes, RGB (Red, Green, and Blue). The use of a single sensor provides an economical and practical way to obtain the three primary colors from an image. In order to capture the three color intensities on a single sensor, a color filter array is used to break the sensor into a mosaic of red, green and blue pixels, as illustrated in FIG. 1. In such a case, the original image is captured with each raw image pixel composed of only one primary color intensity component: either R, G, or B. However, for rendering display, it is desirable to recover a full-color image, where each pixel is composed of a combination of R, G and B color components, from the raw image. Methods to recover full-color images are commonly referred as demosaicing.
A demosaic operation converts a raw image acquired with a single sensor array, overlaid with a color filter array, into a full-color image. “Demosaicing” typically involves interpolating missing color components for a pixel by estimating their values from neighboring pixels. Many demosaic process are specifically targeted at a class of CFA patterns known as Bayer patterns with RGB color space. FIG. 1 is an illustration of a Bayer pattern. This pattern alternates a row of B and G filters with a row of R and G filters. The Bayer pattern makes use of the fact that the human eye is more sensitive to green, and thus more surface of the CFA is used for this color to represent high-frequency detail. The combination of the primary color R, G and B is sufficient to reproduce most colors for visual perception.
Other CFA patterns can also be used to filter light on a sensor. Another popular CFA is the CMYG filter, in which cyan (C), magenta (M), yellow (Y), and green (G) filters are used. FIG. 2 is an illustration of a CMYG CFA pattern.
Many conventional demosaicing approaches exist for converting a raw image data obtained from an RGB Bayer CFA pattern to a full-color image. A simple demosaicing process involves assigning the value of the nearest pixel (any one of the upper, lower, left or right pixel) in the input image as the missing color components. Another approach known as bilinear interpolation involves averaging surrounding pixels to obtain the missing color components for each pixel location. For example, to interpolate green pixels, average the upper, lower, left and right pixel values; thus, according to FIG. 1, the green value for pixel 8 (G8)=(G3+G13+G7+G9)/4. Missing values for red and blue pixels are similarly estimated as linear combinations of available red and blue sensor responses, respectively. However, these approaches are typically prone to undesirable color edge artifacts such as blurring.
Methods to improve color edge artifacts use adaptive color plane interpolation. In one such prior art method used for RGB. Bayer CFA, a set of gradients is determined from the color values in a 5×5 neighborhood centered at the pixel under consideration. Each gradient corresponds to a different direction. For each set of gradients, a threshold value is determined, and the threshold is used to select a subset of gradients. Low-valued gradients indicate pixels having similar color values whereas high-valued gradients would be expected in regions of image where there are many fine details or sharp edges. The subset of gradients is used to locate regions of pixels that are most like the pixel under consideration. The pixels in the region are then weighted and summed to determine the average difference between the color of the actual measured center pixel value and the missing color.
The above adaptive demosaic approach offers good results for sharp edges for RGB Bayer CFA. However, to apply the above demosaic approach to a non-RGB Bayer pattern, the pattern would need to be first converted to an RGB Bayer input pattern. This requires added computational complexity. Thus, there is a need for an edge adaptive demosaic system and method that offers improved edge sharpness and less false color for RGB and non-RGB Bayer CFA input patterns and that does not require conversion to an RGB color space. In particular, there is a need for an edge adaptive demosaic operation that inherently supports CMYG CFA input patterns.